Combining Variational Autoencoders and Physical Bias for Improved Microscopy Data Analysis

Electron and scanning probe microscopy produce vast amounts of data in the form of images or hyperspectral data, such as EELS or 4D STEM, that contain information on a wide range of structural, physical, and chemical properties of materials. To extract valuable insights from these data, it is crucial to identify physically separate regions in the data, such as phases, ferroic variants, and boundaries between them. In order to derive an easily interpretable feature analysis, combining with well-defined boundaries in a principled and unsupervised manner, here we present a physics augmented machine learning method which combines the capability of Variational Autoencoders to disentangle factors of variability within the data and the physics driven loss function that seeks to minimize the total length of the discontinuities in images corresponding to latent representations. Our method is applied to various materials, including NiO-LSMO, BiFeO3, and graphene. The results demonstrate the effectiveness of our approach in extracting meaningful information from large volumes of imaging data. The fully notebook containing implementation of the code and analysis workflow is available at https://github.com/arpanbiswas52/PaperNotebooksComment: 20 pages, 7 figures in main text, 4 figures in Supp Ma

Physics discovery in nanoplasmonic systems via autonomous experiments in Scanning Transmission Electron Microscopy

Physics-driven discovery in an autonomous experiment has emerged as a dream application of machine learning in physical sciences. Here we develop and experimentally implement a deep kernel learning workflow combining the correlative prediction of the target functional response and its uncertainty from the structure, and physics-based selection of acquisition function, which autonomously guides the navigation of the image space. Compared to classical Bayesian optimization methods, this approach allows to capture the complex spatial features present in the images of realistic materials, and dynamically learn structure-property relationships. In combination with the flexible scalarizer function that allows to ascribe the degree of physical interest to predicted spectra, this enables physical discovery in automated experiment. Here, this approach is illustrated for nanoplasmonic studies of nanoparticles and experimentally implemented in a truly autonomous fashion for bulk- and edge plasmon discovery in MnPS3, a lesser-known beam-sensitive layered 2D material. This approach is universal, can be directly used as-is with any specimen, and is expected to be applicable to any probe-based microscopic techniques including other STEM modalities, Scanning Probe Microscopies, chemical, and optical imaging

Describing condensed matter from atomically resolved imaging data: from structure to generative and causal models

The development of high-resolution imaging methods such as electron and scanning probe microscopy and atomic probe tomography have provided a wealth of information on structure and functionalities of solids. The availability of this data in turn necessitates development of approaches to derive quantitative physical information, much like the development of scattering methods in the early XX century which have given one of the most powerful tools in condensed matter physics arsenal. Here, we argue that this transition requires adapting classical macroscopic definitions, that can in turn enable fundamentally new opportunities in understanding physics and chemistry. For example, many macroscopic definitions such as symmetry can be introduced locally only in a Bayesian sense, balancing the prior knowledge of materials' physics and experimental data to yield posterior probability distributions. At the same time, a wealth of local data allows fundamentally new approaches for the description of solids based on construction of statistical and physical generative models, akin to Ginzburg-Landau thermodynamic models. Finally, we note that availability of observational data opens pathways towards exploring causal mechanisms underpinning solid structure and functionality

AtomAI: A Deep Learning Framework for Analysis of Image and Spectroscopy Data in (Scanning) Transmission Electron Microscopy and Beyond

AtomAI is an open-source software package bridging instrument-specific Python libraries, deep learning, and simulation tools into a single ecosystem. AtomAI allows direct applications of the deep convolutional neural networks for atomic and mesoscopic image segmentation converting image and spectroscopy data into class-based local descriptors for downstream tasks such as statistical and graph analysis. For atomically-resolved imaging data, the output is types and positions of atomic species, with an option for subsequent refinement. AtomAI further allows the implementation of a broad range of image and spectrum analysis functions, including invariant variational autoencoders (VAEs). The latter consists of VAEs with rotational and (optionally) translational invariance for unsupervised and class-conditioned disentanglement of categorical and continuous data representations. In addition, AtomAI provides utilities for mapping structure-property relationships via im2spec and spec2im type of encoder-decoder models. Finally, AtomAI allows seamless connection to the first principles modeling with a Python interface, including molecular dynamics and density functional theory calculations on the inferred atomic position. While the majority of applications to date were based on atomically resolved electron microscopy, the flexibility of AtomAI allows straightforward extension towards the analysis of mesoscopic imaging data once the labels and feature identification workflows are established/available. The source code and example notebooks are available at https://github.com/pycroscopy/atomai